Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: add type hints for recipes (#496) #502

Closed
wants to merge 1 commit into from

Conversation

Jiehong
Copy link

@Jiehong Jiehong commented Oct 8, 2020

I decided to give this a try with the smallest file I've found, so that we agree on the direction this goes.

Some notes:

  • I did not use automatic generation of types: I wrote them by hand based on my understanding;
  • I used 'A' and 'B' for typevars, instead of 'T' or '_T', because I find them more explicit (I dislike the C++ notation);
  • I checked, and those types should be available once published with nothing to do (to check).

I was thinking of setting up mypy in the CI/CD to check the type validity, what do you think?

@Jiehong Jiehong force-pushed the feature/type_hints branch 2 times, most recently from 134cb4d to 2d91d85 Compare October 8, 2020 08:30
toolz/recipes.py Outdated

def countby(key, seq):
def countby(key: Callable[[A], B], seq: Iterable[A]) -> Dict[B, int]:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Btw, key can also be a key for e.g. indexing into list or a dict or a list of such keys. For example,

In [8]: toolz.countby('a', [{'a': 1, 'b': 2}, {'a': 10, 'b': 2}])
Out[8]: {1: 1, 10: 1}

In [9]: toolz.countby('b', [{'a': 1, 'b': 2}, {'a': 10, 'b': 2}])
Out[9]: {2: 2}

In [10]: toolz.countby(['a', 'b'], [{'a': 1, 'b': 2}, {'a': 10, 'b': 2}])
Out[10]: {(1, 2): 1, (10, 2): 1}

In [11]: toolz.countby(0, [[1, 2], [10, 2]])
Out[11]: {1: 1, 10: 1}

In [12]: toolz.countby(1, [[1, 2], [10, 2]])
Out[12]: {2: 2}

In [13]: toolz.countby([0, 1], [[1, 2], [10, 2]])
Out[13]: {(1, 2): 1, (10, 2): 1}

How should we declare the type of key? Something like Union[KT, List[KT], Callable[[A], B]]? Do you like KT (following how typing types mappings) or K?

This particular pattern comes up a lot, so perhaps we should make an object to use instead of the Union expression. For example, this could show up in the function definition as def countby(key: Key, .... Key isn't very descriptive. Anything better?

Copy link
Author

@Jiehong Jiehong Oct 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then, in this case, seq can be different things too.

It's like we have multiple versions of the same countby, as the type of key seems to be linked to the type of seq.

So, we have the following versions if I understand well:

# Case: countby(len, ['cat', 'mouse', 'dog'])
def countby(key: Callable[[A], B], seq: Iterable[A]) -> Dict[B, int]:

# Case: countby('a', [{'a': 1, 'b': 2}, {'a': 10, 'b': 2}])
def countby(key: K, seq: Iterable[Dict[K, B]]) -> Dict[B, int]:

# Case: countby(['a', 'b'], [{'a': 1, 'b': 2}, {'a': 10, 'b': 2}])
def countby(key: List[K], seq: Iterable[Dict[K, B]]) -> Dict[Tuple[B,...], int]:

# Case: countby(0, [[1, 2], [10, 2]])
def countby(key: int, seq: Iterable[Iterable[A]]) -> Dict[A, int]:

# Case: countby([0, 1], [[1, 2], [10, 2]])
def countby(key: List[int], seq: Iterable[Iterable[A]]) -> Dict[Tuple[A, ...], int]:

So, in this case, the types aren't so useful for such a generic function, because they would loose that dependency between inputs and output.

What do you think?

I do agree on reusing an object instead of duplicating the Union everywhere though.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the end, I think the most generic way I could type this is:

def countby(key: KeyLike, seq: Iterable) -> Dict[Any, int]:

Python's type system does not allow making the dependency possible.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, perhaps I should use Sequence instead of Iterable. What do you think?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can have dependencies if you use overload. Something like,

# Case: countby(len, ['cat', 'mouse', 'dog'])
@overload
def countby(key: Callable[[A], B], seq: Iterable[A]) -> Dict[B, int]: ...

# Case: countby('a', [{'a': 1, 'b': 2}, {'a': 10, 'b': 2}])
@overload
def countby(key: K, seq: Iterable[Dict[K, B]]) -> Dict[B, int]: ...

def countby(key: KeyLike, seq: Iterable) -> Dict[Any, int]: 
   # real definition

The mypy documentation discusses overload here.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I've updated that.

@eriknw
Copy link
Member

eriknw commented Oct 9, 2020

I like where this is going; thanks @Jiehong!

I like how you used A and B. I'm all for setting up mypy check in CI.

Let's keep going!

@Jiehong Jiehong force-pushed the feature/type_hints branch from 2d91d85 to f1d10fd Compare October 9, 2020 13:22
@Jiehong
Copy link
Author

Jiehong commented Oct 9, 2020

I've updated the PR with mypy running on recipes.py only. I had to ignore some issues in some lines by adding some comments to make it pass.

Mypy also helped choosing between Iterable and Sequence fairly easily :D

.travis.yml Outdated
@@ -32,6 +32,7 @@ script:
- python setup.py develop
- py.test
- nosetests
- mypy toolz/recipes.py
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only checking the file I modified added type hints to. But it depends on other files: this explains why I had to ignore some errors / make mypy happy elsewhere.

@Jiehong Jiehong force-pushed the feature/type_hints branch from f1d10fd to 0f3cc15 Compare October 9, 2020 13:28
@Jiehong
Copy link
Author

Jiehong commented Oct 9, 2020

pypy seems to have issues with mypy installation, somehow.

@Jiehong Jiehong force-pushed the feature/type_hints branch 5 times, most recently from 079a165 to 5abba72 Compare October 11, 2020 12:05
@Jiehong
Copy link
Author

Jiehong commented Oct 11, 2020

@eriknw : this PR is now ready for review:

  • mypy is setup (but only when not using pypy, as the installation fails);
  • mypy reports no errors;
  • recipes.py is correctly typed.

If you're fine with this PR, then it can be merged.

I'll make new PRs for another file only after that. I want to avoid make 1 huge PR to add types everywhere at once.

@Jiehong Jiehong force-pushed the feature/type_hints branch from 5abba72 to 646ef40 Compare March 12, 2021 21:09
@Jiehong Jiehong closed this Jun 16, 2022
@getzze getzze mentioned this pull request Mar 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants